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1.0  Introduction 

In  this  report  we  describe  our  work  on  developing  improved  computer  vision  systems  for  aerial 
scenes.  Our  research  efforts  included  research  on  developing  a  hierarchical  vision  system  for  aerial 
scenes,  comprehensive  vision  operators,  detection  of  linear  features,  developing  theory  for  defining  new 
measures  which  reflect  perceptual  properties,  developing  operators  and  measures  for  characterizing  tex¬ 
ture  patterns  and  evaluating  the  theory  on  aerial  scenes. 

A  computer  vision  system  can  be  conceptualized  as  being  composed  of  vision  operators,  an 
inferencing  system,  and  a  knowledge  base  of  models  and  facts.  The  vision  system  should  label  and 
describe  the  objects,  subobjects  and  relationships  in  the  scene.  The  vision  operators  are  used  to  extract 
image  cues  from  the  image.  The  role  of  the  inferencing  system  is  to  sequence  or  select  the  models  in 
the  knowledge  base,  match  the  extracted  image  cues  against  the  models,  resolve  conflicts  and  track  the 
inferences.  In  creating  a  vision  system  a  number  of  problems  must  be  addressed.  These  are:  the 
interaction  between  the  vision  operators  and  the  inferencing  system,  the  selection  and  adequacy  of  the 
vision  operators,  the  method  of  forming  an  optimal  global  interpretation  of  the  scene,  and  the  integra¬ 
tion  of  system  modules  and  data  structures  so  that  the  system  can  be  improved  as  additional  knowledge 
about  the  scene  becomes  known. 

The  quality  of  the  measurements  or  image  cues  obtained  by  the  vision  operators  is  a  key  to  sys¬ 
tem  performance.  If  the  operators  return  reliable  information  with  a  rich  descriptive  vocabulary,  then 
the  vision  system  is  greatly  enhanced. 

2.  General  Purpose  Operators 

There  are  three  important  issues  related  to  operators  for  a  computer  vision  system.  The  first  is  that 
the  operators  must  be  widely  applicable  or  general  purpose;  secondly  the  operators  must  have  a  rich 
descriptive  vocabulary;  thirdly  the  operators  must  be  suitable  for  obtaining  image  cues  from  intermedi¬ 
ate  levels  of  the  scene  hierarchy.  We  feel  that  the  approach  to  be  pursued  first  is  to  model  Gestalt 
principles  of  perceptual  organization.  These  approaches  can  aid  in  building  a  geometric  reasoning  sys¬ 
tem.  we  are  interested  in  developing  general  purpose  operators  which  measure  perceptual  properties. 
These  operators  offer  advantages  in  their  descriptive  vocabulary  over  traditional  operators.  They  can 
also  be  utilized  at  intermediate  levels  of  the  scene  hierarchy. 

The  present  capabilities  of  texture  algorithms  severely  limit  there  usefulness  in  vision  systems. 
Present  algorithms  utilize  supervised  training  methods  to  select  measures.  Many  times  the  measures 
selected  may  be  correlated  and  may  not  even  be  measuring  relevant  information  about  the  scene  struc¬ 
ture.  One  needs  a  higher  level  of  proficiency  in  texture  algorithms.  One  would  like  to  avoid  super¬ 
vised  learning.  One  desires  a  metric  to  correctly  gauge  the  proximity  of  measurements  in  measurement 
space  when  the  areas  from  which  the  measurements  are  obtained  are  close  perceptually.  Theoretical 
results  directed  towards  solving  this  problem  are  reported  in  Vasquez[1984].  The  solution  to  this  prob¬ 
lem  would  allow  one  to  obtain  better  results  utilizing  clustering  or  region  growing  methods  to  segment 
the  image.  At  the  higher  levels  of  analysis  this  would  ensure  the  vision  system  is  robust.  For  exam¬ 
ple,  residential  scenes  will  all  vary  to  some  extent.  It  is  required  to  have  operators  which  reflect  the  per¬ 
ceptual  properties  of  these  areas  in  order  to  label  the  different  areas  correctly.  The  most  desirable  situa¬ 
tion  would  be  for  operators  to  characterize  scenes  using  the  same  perceptual  properties  utilized  by 
humans.  Examples  of  such  properties  would  be  symmetry  [Man  1982,  Julesz  1969],  periodicity[Mach 
1959],  uniformity[Wertheimer  1958],  edge  detection  and  comer  detection  [Hubei  and  Wiesel  1968]. 
This  would  be  a  step  in  the  direction  of  providing  a  richer  descriptive  vocabulary  for  the  operators 
related  to  human  perceptual  properties,  if  these  operators  could  be  developed,  they  would  have  great 
utility  in  vision  systems. 

It  is  our  belief  that  these  operators  can  be  based  upon  algorithms  previously  applied  in  various 
forms  to  texture  patterns.  We  will  show  later  in  this  report  that  the  operators  have  broader  application 
than  to  texture  patterns, 

3.  Theory  of  Measurement  Definition. 


The  framework  is  based  on  the  concepts  of  a  perceptual  transform,  a  perceptual  space,  a  measure¬ 
ment  space,  a  measurement  transform,  a  measurement  space,  and  a  similarity  transform  relating  how 
similar  the  measurement  space  and  the  perceptual  space  must  be. 

Let  W  be  the  set  of  all  possible  textures  and  let  w  denote  an  element  of  W. 

Definition  1.  The  perceptual  transform,  Tp(w)  is  an  n-dimensional  vector  of  visual  features  "com¬ 
puted"  by  the  human  perceptual  system.  It  maps  the  set  W  into  the  perceptual  space,  Ps. 

Definition  2.  Two  textures  wl  and  w2  are  visually  distinct  if  and  only  if  Tp(wl)  is  not  equal  to 
Tp(w2),  that  is,  if  wl  and  w2  are  mapped  into  different  points  in  the  perceptual  space,  Ps. 

Definition  3.  A  measurement  to  transform,  Tm  represents  the  k-dimensional  vector  of  measures 
computed  from  a  texture  where  each  component  of  Tm  is  a  texture  measure.  Tm  maps  elements  of  W 
into  a  k-dimensional  measurement  space,  Ms. 

Definition  4.  A  similarity  transform,  Ts  maps  the  points  of  Ps  into  the  points  of  Ms.  Definition  5. 
The  set  of  measures  defined  by  Tm(w)  is  said  to  match  with  similarity  Ts  the  human  perceptual  abil¬ 
ity  if  Ts(Tp(w))  =  Ttn(w))  for  all  w  in  W. 

This  definition  establishes  the  goal  a  measurement  definition  problem,  namely,  given  Ts,  define 
the  measurements  such  that: 


Ts(Tp(w))  =  Tm(w).  (1.) 

for  all  w  in  W. 

Theoretically,  the  "best”  set  of  measures  would  seem  to  be  those  defined  when  Ts  is  the  identity 
transform.  This  would  force  the  texture  measures  to  be  the  same  as  the  features  used  by  the  human 
perceptual  system. 

There  are,  however,  two  good  arguments  against  attempting  this.  First,  what  may  be  optimal  for 
humans,  computationally,  may  be  very  suboptimal  for  computers.  Secondly,  the  present  state  of  percep¬ 
tual  psychology  and  neurophysiology  makes  such  an  objective  practically  impossible. 

Rather,  the  appropriate  course  of  action  would  seem  to  be  to  determine  in  what  ways  we  would 
like  our  early  vision  operators  to  mimic  human  performance.  Once  this  is  done,  one  can  define  a  Ts 
which  captures  these  essential  characteristics  so  that  an  appropriate  set  of  measures  can  be  identified. 

Let  us  present  the  ways  we  believe  texture  operators  should  mimic  human  perceptual  perfor¬ 
mance. 

1.  Texture  measures  should  be  able  to  discriminate  any  two  visually  distinct  textures. 

2.  The  texture  measures  should  be  such  that  a  norm  on  the  measurement  space  will  preserve  human 

judgements  about  the  similarity  of  textures.  This  comes  from  the  realization  that  one  of  the  most 

important  task  of  any  early  vision  operator  is  to  aid  in  segmenting  the  image. 

3.  The  dimensionality  of  the  measurement  space  should  be  as  small  as  possible. 

These  desires  translate  into  a  set  of  requirements  on  Ts.  These  requirements  are  as  follows: 

1.  Ts  should  be  one-to-one. 

2.  If  wl  and  w2  are  any  two  textures  in  W  then: 

N(Tm(wl),Tm(w2))  -  Np(Tp(wl),Tp(w2)).  (2.) 

where  N  is  a  norm  on  the  measurement  space  and  Np  is  a  norm  on  the  perceptual  space. 

3.  For  each  measurement  mi,  a  component  of  Tm,  there  should  be  two  visually  distinct  textures,  say 
wl  and  w2,  such  that  mi(wl)  is  not  equal  to  mi(w2). 

A  quick  analysis  will  show  that  requirements  1  and  2  on  Ts  are  redundant  since  requirement  2 

insures  requirement  1.  Hence,  there  is  really  only  one  requirement  on  Ts  to  drive  the  definition 


process.  The  key  to  using  this  requirement  is  having  a  perceptual  norm,  Np,  that  for  a  set  of  texture 
pairs  can  give  a  relative  ranking  as  to  how  visually  distinct  each  pair  is  as  compared  to  the  others.  One 
such  ranking  scheme  is  the  law  of  comparative  judgement  (LCI)  developed  by  Thurston  [1927). 

Given  this  perceptual  ranking  scheme,  assume  we  have  s  texture  pairs,  pi  -  (wll,w2i),  i-1 . s.  A 

perceptual  experiment  can  be  performed  [Zobrist  and  Thompson  1975),  so  that  the  law  of  comparative 
judgement  can  be  used  to  give  a  ranking  rpi  to  each  of  the  texture  pairs  pi,  where  each  rpi,  i-l...s,  is  a 
positive  real  number. 

For  each  of  the  s  texture  pairs,  requirement  2  demands  that 

N(Tm(wli),Tm(w2i))  =  rpi.  (3.) 

However  it  is  unlikely  that  any  measurement  set  can  be  defined  so  that  Eq.  3  is  exacdy  satisfied  for 
i=l,....,s.  Rather  it  seems  more  probable  that  for  each  possible  measurement  set  Tm(w)  we  will  have 

N(Tm(wli),Tm(w2i))  -  rpi  +  ei.  (4.) 

where  ei  represents  an  error  term.  Therefore,  the  more  reasonable  approach  is  to  pick  the  measurement 
set  which  minimizes  the  sum  of  squares  of  the  error  terms,  i.e.,  a  least-squares  fit 

To  integrate  requirement  3  into  the  definition  process  requires  the  following  procedure  be  fol¬ 
lowed  in  defining  the  measures.  First  solve  for  one  measure.  You  do  not  solve  for  a  second  measure¬ 
ment  until  a  visually  distinct  texture  pair  has  been  found  that  cannot  be  discriminated  by  this  one  meas¬ 
ure.  Given  the  existence  of  such  a  texture,  one  then  solves  for  two  measures,  etc. 

To  use  this  formal  definition  process  requires  (a)  a  perceptual  ranking  experiment  be  performed; 
(b)  the  LCJ  be  applied  to  the  experimental  data;  (c)  a  least  squares  procedure  (LSP)  be  implemented  to 
solve  for  the  measures;  (d)  an  assumption  about  the  form  for  the  norm,  N,  on  the  measurement  space, 
and  about  a  functional  form  for  the  measurements;  and  (e)  a  good  texture  synthesis  procedure  for  gen¬ 
erating  texture  pairs  used  both  in  the  experiment  and  in  establishing  whether  additional  measures  are 
needed. 

Given  the  assumptions  required  to  perform  the  least  squares  analysis,  this  measurement  definition 
process  assures  that  the  measurements  defined  are  such  that  there  exists  a  transformation  Ts  in  the  class 
of  transformations  satisfying  the  above  requirements  where 

Ts(Tp(w)  -  Tm(w).  (5.) 

for  all  textures  w  comprising  the  texture  pairs  used  in  the  perceptual  experiment.  If  these  textures  are 
representative  of  the  universe  of  textures,  then  the  goal  of  the  measurement  definition  process  as  given 
in  Eq.  2  is  satisfied. 

Using  this  Formal  Method  on  a  Simplified  Problem 

To  test  this  formal  definition  process,  it  was  employed  on  a  simplified  problem.  The 
simplifications  were  that  the  texture  pain  used  in  the  perceptual  ranking  experiment  were  generated 
using  only  one  texture  synthesis  method  and  that  these  textures  contained  only  three  gray  levels.  Limit¬ 
ing  the  number  of  gray  levels  markedly  reduces  the  number  of  texture  pairs  that  must  be  considered  to 
solve  for  the  measurements.  Using  only  one  synthesis  method  provided  other  simplifications  useful  in 
the  least-squares  fitting  of  the  data. 

The  experimental  procedure  used  with  the  law  of  comparative  judgement  was  the  method  of 
paired  comparisons.  This  experimental  procedure  requires  that  each  stimulus  (texture  pair)  be  compared 
with  all  the  other  stimuli.  Hence  to  perceptuaily  rank  the  twenty-two  texture  pairs  requires  231  paired 
comparisons  be  made. 

To  make  these  comparisons,  231  display  boards  were  prepared.  The  231  boards  were  presented 
to  18  different  observers,  each  observer  seeing  the  boards  on  two  occasions  with  the  order  of  presenta¬ 
tion  being  reserved  on  the  second  viewing  by  each  observer.  The  question  asked  was  "which  of  these 
two  texture  pairs  contains  the  more  visually  distinct  textures?" 

Since  the  results  when  only  one  measure  was  defined  were  unsatisfactory,  we  solved  for  two 
measures.  The  residual  plot  indicates  an  improvement  but  indicates  that  still  more  measures  might  be 


-6- 


necessary.  We  call  these  measures  M12  and  M22.  Attempts  were  made  to  solve  for  three  and  then  four 
measurements.  However  no  marked  improvement  in  the  quality  of  fit  occurred.  The  reason  for  this  can 
be  explained  by  making  two  observations.  First  every  measure  that  is  defined  by  this  method  requires 
3x3  »  9  coefficients  be  estimated.  Thus  when  one  attempts  to  solve  for  first  one,  then  two,  then  three 
and  then  four  measures,  9,  18,  27,  and  then  36  coefficients  must  estimated,  respectively.  Since  there 
are  only  22  different  texture  pairs  whose  relative  rankings  are  being  used  to  drive  the  estimation  pro¬ 
cess,  attempting  to  solve  for  more  than  two  measures  makes  mote  unknowns  than  there  are  equations 
constraining  these  unkn  jwns.  Thus,  it  seems  reasonable  that  no  improvement  is  noted. 

It  is  of  interest  to  note  that  the  measures,  M12  and  M22,  defined  using  the  formal  measurement 
definition  process,  appear  functionally  equivalent  to  the  contrast  measure  and  symmetry  measure  whose 
definitions  arose  from  the  consideration  of  periodic  textures.  That  is,  measure  M12  can  gauge  the  pres¬ 
ence  or  absence  of  nonzero  off  diagonal  elements,  the  job  the  contrast  measure  was  picked  to  do  in  the 
analysis  of  periodic  textures.  Further,  measure  M22  can  be  used  to  gauge  the  symmetry  or  asymmetry 
of  a  GLC  matrix.  This  was  demonstrated  by  comparing  the  spectnuns  of  measure  MU  and  the  con¬ 
trast  measure  on  a  number  of  images.  Both  clearly  indicate  periodic  structure  of  the  texture  patterns  in 
the  images.  In  a  like  manner  the  spectrums  of  the  symmetry  measure  and  a  simple  function  of  M22  also 
appear  very  similar,  each  indicating  the  direction  of  maximal  symmetry  and  asymmetry  of  the  pattern  in 
the  image.  The  simple  functional  for  of  M22  which  is  used  is 

f(M22(d,T))  -  1  M22(d,T)  -  M22(-d,T)  L 

In  summary,  the  primitive  measures  defined  using  two  completely  different  approaches  to  the 
definition  problem  appear  functionally  equivalent.  This  suggests  the  validity  of  die  theoretical 
approach  to  measurement  definition.  It  is  also  interesting  to  note  that  these  measures  appear  to  form  the 
basis  for  truly  general  purpose  operators. 

4.  GLC  Matrices  and  Tiling  Theory. 

For  notational  convenience  we  often  let  u  -  (x,y)  denote  a  two  dimensional  vector  in  E 2  the 
Euclidean  plane. 

Definition  1.  A  die  T  is  a  subset  of  the  Euclidean  plane,  £2,  that  is  a  closed  topological  disk. 

Definition  2.  A  plane  filing  is  a  family  of  tiles  T  -  {T,  1  i-1,2,...}  that  covers  the  plane  without  holes  or 
overlaps.  The  tiling  is  monohedral  if  each  tile  in  the  tiling  is  a  single  prototile. 

Definition  3.  A  function  s  that  maps  £2  into  £ 2  is  called  an  isometry  or  congruence  transformation  if 
it  maps  £2  onto  itself  and  if  x,  y  are  elements  of  £2  then  N(x,y)  -  N(s(x),  s(y))  where  N  is  the  standard 
Euclidean  norm.  That  is,  an  isometry  is  a  mapping  which  preserves  distance. 

An  isometry  is  a  symmetry  of  a  tiling  if  it  maps  every  tile  of  a  tiling  into  another  file  of  the  til¬ 
ing.  If  there  exist  at  least  two  translation  symmetries  in  nonparallel  directions,  then  the  tiling  is  said  to 
be  periodic.  Let  these  directions  be  given  by  unit  vectors  a  and  b.  The  set  of  all  translations  {na  + 
mb}  forms  a  lattice  which  give  the  vertices  of  a  parallelogram  tiling. 

Definition  4.  The  tiles  of  the  parallelogram  filing  are  known  as  the  period  parallelogram. 

These  concepts  are  extended  to  texture  patterns  by  considering  the  gray  levels  within  the  tile. 

Definition  5.  A  primitive  is  {T,f}  where  T  is  a  file  and  f  is  a  function  that  maps  T  into  the  nonnegafive 
real  numbers. 

We  call  f  a  painting  function  which  gives  the  tile  a  pattern. 

Definition  6.  A  set  of  primitives  {T 1/2},...,}  {TKjm} . .  is  admissible  if  the  set  of  prototiles 

{TuT2 . T„}  admits  a  tiling  of  the  plane. 

Each  element  {7}/, }  is  called  a  unit  pattern.  T,  is  called  a  unit  file. 


Definition  7.  A  gray  level  cooccurrence  (GLC)  matrix,  S(d,T),  is  a  matrix  of  estimated  second-order 
probabilities  where  each  element  S(ij,d,T)  is  defined  by 

SVJ4.T 

0 (u  I u,u+dtT) 


t 


where  T  is  a  tile,  S(ij,d,T)  is  estimated  from  the  restriction  of  the  picture  function  g(x)  to  T,  and  where 
0  denotes  the  order  of  the  set. 

It  is  frequently  convenient  to  consider  d  =  (5x,8y)  not  in  cartesian  form  but  rather  in  a  polar  form 
d  -  (r,0)  where  r  =  max(8x,8y)  and  0  -  arc  tan  (8y/8x). 

Definition  8.  A  primitive  measure,  M{d,T)  is  a  real  valued  function  whose  independent  variables  are 
the  elements  of  a  GLC  matrix,  S(d,T). 

An  example  of  a  primitive  measure  is  the  contrast  measure,  C(d,T),  defined  by 


C  (d  ,T)=L  2-±  £(i-y  )2S  (i  J  4  ,T) 

i«0/s0 

where  L  is  the  number  of  gray  levels.  The  contrast  measure  varies  from  0  to  L2. 

Definition  9.  The  spectrum,  Q(M,T,R),  of  the  primitive  measure  M(d,T)  is  M(d,T)  evaluated  for  all  d 
in  R  where  R  is  a  region  of  E 2  containing  the  origin. 

For  example,  the  circular  spectrum  of  the  contrast  measure  C(d,T),  is  C(d,T)  evaluated  for  all  d  in 
R  where  R  is  the  set  of  all  points  (Sr,8y)'  such  that  &x2+8y2Sr2. 

Definition  10.  An  spectral  measure,  K(M,T,R),  is  a  real  valued  function  whose  independent  variables 
are  the  elements  of  the  spectrum  Q(M,T,R)  where  M(d,T)  is  a  primitive  measure. 

No  single  GLC  matrix  can  characterize  a  texture.  Rather,  a  number  of  GLC  matrices  is  required. 
This  means  that  to  characterize  the  structure  of  an  image  one  will  need  spectral  measures  derived  from 
the  spectra  of  primitive  measures. 

5.  Periodicity  Measures. 

One  set  of  patterns  that  one  would  desire  to  analyze  are  periodic  textures.  Our  work  on  periodic 
textures  motivated  the  formulation  of  a  tiling  theory  model  of  texture[Conners  and  Harlow  1980].  One 
should  also  note  that  periodicity  is  a  perceptual  property  readily  perceived  by  humans.  In  Conners  and 
Harlow  [1980]  it  was  shown  that  to  characterize  periodic  textures  one  need  only  determine  a  period 
parallelogram  unit  pattern.  This  requires  that  one  determine  two  vectors  a  and  b  which  specify  the 
period  parallelogram  unit  pattern.  Figure  1. 

The  GLC  matrices  can  detect  periodicity.  Suppose  in  direction  d  the  pattern  has  periodicity  t  If  d 
is  a  unit  vector  then  one  has  the  following  properties. 

L  The  matrices  S(nd,T)  for  n  =  1,2,  ....  t-1  have  nonzero  off  diagonal  elements. 

2.  The  matrices  S(ntd,T)  for  n  =1,2,...  are  diagonal  matrices  with  no  nonzero  off  diagonal  elements. 

3.  The  GLC  matrices  are  periodic  in  direction  d  with  periodicity  t.  That  is,  S((nt+m)d,T)  =  S(md,T) 

for  m,n=  1,2 . 

This  implies  that  a  measure  such  as  the  contrast  measure  should  reflect  the  periodicity  of  patterns. 
The  contrast  measure  should  be  zero  when  the  GLC  matrix  is  diagonal  and  nonzero  otherwise. 

The  complete  analysis  of  a  periodic  patterns  requires  determining  the  size  and  shape  of  the  unit 
tile  as  was  done  above.  In  addition,  the  analysis  requires  determining  the  painting  function  of  the  unit 
cell.  This  means  the  correct  placement  of  the  unit  cell  must  be  determined.  We  feel  the  one  humans 


i 


would  select  is  the  placement  where  the  object  is  centered  in  the  cell.  This  is  equivalent  to  placing  the 
unit  die  so  that  the  pattern  is  symmetrical.  One  might  note  that  symmetry  is  a  perceptual 
property[Julesz  1969]  and  psychologists  have  long  thought  the  property  to  be  important  in  figure/ground 
determinadon  where  the  symmetrical  pattern  would  be  the  figure.  Therefore,  the  problem  can  be 
reduced  to  locating  a  symmetrical  pattern  which  will  then  completely  determine  {T,f}  where  T  is  the 
parallelogram  tile  and  f  is  the  paindng  funcdon.  The  GLC  matrices  can  be  utilized  to  determine  sym¬ 
metry. 


6.  Symmetry  Measures. 


Definition  1.  A  function  f  defined  on  a  tile  T  is  symmetric  about  the  x  axis  if: 

1.  If  (x,y)  e  T  then  (-x,y)  e  T  and 

2.  if  (x,y)  e  T  then  f(x,y)  =  f(-x,y). 

Definition  2.  A  function  f  defined  on  a  tile  T  is  symmetric  about  the  origin  if: 

1.  If  (x,y)  e  T  then  (-x,-y)  e  T  and 

2.  if  (x,y)  e  T  then  f(x,y)  =  f(-x,-y). 

The  coordinate  system  to  which  these  definitions  refer  need  not  be  orthogonal  but  can  be  a  local 
coordinate  system  defined  on  the  period  parallelogram,  Figure  1.  In  this  local  coordinate  system  the  x 
axis  is  aligned  with  the  a  vector  and  the  y  axis  is  aligned  with  the  b  vector.  This  is  the  coordinate  sys¬ 
tem  of  interest  in  the  following  results. 

Result  1.  If  a  function  defined  on  a  tile  T  is  symmetric  about  both  the  x  and  y  axis,  then  it  is  sym¬ 
metric  about  the  origin.  The  converse  is  not  necessarily  true. 

Let  us  now  relate  these  concepts  to  GLC  matrices.  In  the  following  results  we  assume  that 
{u  |  u,u+d  e  T}  is  not  the  empty  set. 

Result  2.  If  a  function  f  defined  on  a  tile  T  is  symmetric  about  the  x  axis  of  a  local  coordinate  sys¬ 
tem,  then  the  GLC  matrix  S(d,T)  is  a  symmetric  matrix  when  d,  defined  in  the  local  coordinate  system, 
is  equal  to  (0,5y). 

Result  3.  If  a  function  f  defined  on  a  tile  T  is  symmetric  about  the  y  axis  of  a  local  coordinate  sys¬ 
tem,  then  the  GLC  matrix  S(d,T)  is  a  symmetric  matrix  when  d,  defined  in  the  local  coordinate  system, 
is  equal  to  (8x,0). 

Result  4.  If  a  function  f  defined  on  a  tile  T  is  symmetric  about  the  origin  of  a  local  coordinate  sys¬ 
tem,  then  the  GLC  matrix  S(d,T)  is  a  symmetric  matrix  when  d  is  defined  in  the  local  coordinate  sys¬ 
tem. 

A  typical  GLC  matrix  is  not  symmetric.  This  then  leads  us  to  the  following  hypotheses: 

Hypothesis  1.  If  a  function  is  defined  on  a  parallelogram  tile  T  is  asymmetric  about  the  x  axis  of  the 
local  coordinate  system  aligned  with  the  parallelogram,  then  there  exists  a  displacement  d  of  the  form 
d»(0,8y)  where  S(d,T)  is  an  asymmetric  matrix. 

Hypothesis  2.  If  a  function  is  defined  on  a  parallelogram  tile  T  is  asymmetric  about  the  y  axis  of  the 
local  coordinate  system  aligned  with  the  parallelogram,  then  there  exists  a  displacement  d  of  the  form 
d=(8x,0)  where  S(d,T)  is  an  asymmetric  matrix. 

Hypothesis  3.  If  a  function  defined  on  a  parallelogram  tile  T  is  asymmetric  about  the  origin  of  the 
local  coordinate  system  aligned  with  the  parallelogram,  then  there  exists  a  d  defined  in  terms  of  this 
same  coordinate  system  such  that  S(d,T)  is  an  asymmetric  matrix. 


Experimental  results  obtained  so  far  indicates  that  these  hypothesis  are  true  for  data  obtained  from 
images.  The  following  results  have  been  shown  analytically  to  be  true. 

Result  5.  Suppose  f  is  a  function  defined  on  a  parallelogram  unit  tile  T  specified  by  vectors  a  and  b 
and  the  unit  pattern  {T,f}  is  asymmetric  about  the  x  axis  of  the  coordinate  system.  Let 

{(*i0'i).(*20’2).— (*«»>«)}  be  a  set  of  points  such  that  ffr.-.y,)  is  not  equal  to  ffo.-y,).  Without  loss  of 

generality  let  (xi.yi)  be  such  that  yt>y,-  for  i-  2,...,n.  Also  Ocy i^J|b||/2  where  b  is  the  vector  parallel  to 
the  y  axis  in  the  parallelogram  coordinate  system.  Then  if  d=(0,yi+(||b||/2)-l),  the  GLC  matrix  S(d,T)  is 
an  asymmetric  matrix. 

There  is  a  similar  result  for  the  y  axis. 

Result  6.  Suppose  f  is  a  function  defined  on  a  parallelogram  unit  tile  T  specified  by  vectors  a  and  b 
and  the  unit  pattern  {T,f}  is  asymmetric  about  the  y  axis  of  the  coordinate  system.  Let 

{(*iO'i)*(*2»y2)»-**(J1»*yii)}  be  a  set  of  points  such  that  f(x,,y,  )  is  not  equal  to  f(-x, ,y, ).  Without  loss  of 

generality  let  (x  x,y  j)  be  such  that  xj>x(  for  i«  2,...,n.  Also  (kx^lbll/2  where  b  is  the  vector  parallel  to 
the  y  axis  in  the  parallelogram  coordinate  system.  Then  if  d=(0rr,+(Hb|l/2)-l),  the  GLC  matrix  S(d,T)  is 
an  asymmetric  matrix. 

These  results  indicate  how  the  analysis  might  proceed  to  determine  the  correct  location  of  the  unit 
tile  might  proceed.  First,  one  must  check  GLC  matrices  to  determine  if  they  are  symmetric.  In  addi¬ 
tion,  one  must  consider  d  of  the  form  (8x,0)  and  (0,8y),  where  0<8x£||a||-  1  and  Ck8y<||b||  -  1  where  d 
is  given  in  the  local  coordinate  system  aligned  with  vectors  a  and  b. 

The  above  discussion  indicates  a  need  for  a  measure  to  characterize  the  symmetry  of  a  pattern  in 
direction  8  where  0=arctan8y/8x  when  d=(8x,8y).  A  measure  which  characterizes  the  symmetry  of  a 
GLC  matrix  is  given  by 

U{d ,7>2-££  1 5  (i ,j ,d,T)-S  O' ,i ,d ,T)  \ . 

where  L  is  the  number  of  gray  levels.  We  call  U  the  symmetry  measure.  U  is  large  when  the  pattern  is 
symmetrical  in  direction  0  and  U  is  small  when  the  pattern  is  asymmetrical  in  direction  8. 

7.  Global  and  Local  Analysis  of  Periodic  Texture  Patterns. 

Let  us  now  consider  the  utilization  of  these  concepts  to  analyze  textured  objects. 

First  consider  the  tank  farm  scene  in  Figure  2.  The  initial  step  is  to  compute  the  contrast  spec¬ 
trum  as  shown  in  Figure  2.  One  then  determines  the  local  maxima  in  the  contrast  spectrum.  We  find  the 
position  and  value  of  these  maxima.  The  periodicities  indicate  the  vectors  a  and  b.  Clearly,  in  this 
example  they  are  in  the  horizontal  and  vertical  directions.  A  tiling  is  then  placed  on  the  area  which 
composes  an  edge  to  edge  tiling  of  the  area.  The  correct  placement  of  the  tiling  remains  to  be  deter¬ 
mined.  The  placement  of  the  tiling  will  specify  the  painting  function  of  the  unit  tile.  This  is  defined  to 
be  the  location  that  creates  a  symmetric  pattern  in  the  unit  tile.  Next,  we  compute  the  symmetry  spec¬ 
trum  which  is  computed  using  the  symmetry  measure.  For  each  x,y  position  in  the  a  vector  direction 
and  for  each  x,y  position  in  the  b  vector  direction  one  computes  the  symmetry  measure  as  indicated  in 
Result  6  of  section  6.  Bar  measures  in  the  a  and  b  are  computed  on  the  symmetry  spectrum.  These 
measures  will  be  a  maximum  in  the  desired  position.  The  symmetry  spectrum  can  be  used  to  rank  the 
symmetry  and  thereby  select  the  correct  position  of  the  tile.  The  correct  tile  position  is  selected  as  the 
x,y  position  which  creates  a  symmetrical  pattern.  This  is  shown  in  Figure  2. 

Let  us  now  consider  how  these  operators  might  be  utilized  in  a  vision  system.  This  involves  a 
global  to  local  analysis.  The  global  analysis  would  require  determining  the  vectors  a  and  b  which 
characterize  the  unit  tile  T  and  then  determining  the  correct  location  which  implies  the  painting  func¬ 
tion  f  since  g  the  image  function  restricted  to  T  defines  f.  This  gives  {T,f}  which  characterizes  the  glo¬ 
bal  properties  of  the  pattern.  The  calculation  for  determining  the  unit  die  T  would  proceed  as  indi¬ 
cated  above  udlizing  the  contrast  spectrum.  The  correction  locadon  for  T  would  be  determined  using 
the  symmetry  measure  as  indicated  above.  These  are  examples  of  intermediate  level  measures  which 
could  be  used  in  the  analysis  hierarchy.  Note  that  these  measures  characterize  the  pattern  of  interest 


directly  and  are  not  inferred  from  a  lower  level  in  the  hierarchy.  Once  {T,f}  has  been  determined  the 
search  for  details  would  proceed  to  the  next  level  in  the  hierarchy.  Let  us  now  turn  consideration  to 
detecting  visual  features  of  a  different  type. 

8.  Operators  for  Detecting  Edges  and  Linear  Features. 

Recall  that  in  the  previous  section  we  indicated  how  GLC  operators  could  be  used  to  character¬ 
ized  textured  objects.  This  involved  a  global  analysis  to  determine  the  unit  tile  which  was  determined 
by  vectors  a  and  b.  A  local  analysis  was  then  performed  to  determine  the  painting  function.  It  is 
interesting  to  note  that  a  distributed  pattern  such  as  a  texture  pattern  requires  a  local  analysis  which  is 
the  type  of  analysis  one  expects  to  perform  on  a  local  object  Local  objects  are  not  characterized  by  a 
distributed  texture  pattern.  In  the  past  local  objects  were  usually  characterized  by  an  edge  detection  or 
thresholding  method  which  utilizes  gray  level  differences  or  gray  level  clusters  to  determine  objects  or 
regions  [Rosenfeld  1982,  Riseman  and  Arbid  1977],  In  this  section  we  intend  to  generalize  the  the  con¬ 
cepts  presented  in  the  last  section  to  include  the  ability  to  analyze  edges  and  linear  features.  We  intend 
to  analyze  these  features  when  they  are  characterized  by  texture  pattern  differences  as  well  as  when 
they  are  characterized  by  gray  level  differences. 

The  techniques  used  in  this  development  will  be  bar  measures  extracted  from  the  contrast  and 
symmetry  spectrums.  Recall,  that  the  contrast  and  symmetry  spectre  ms  were  utilized  in  the  previous 
section  to  characterize  texture  patterns.  A  bar  measure  is  a  spectral  measure  which  gives  the  average 
value  over  a  bar  shaped  area  of  a  spectrum.  Figure  3  depicts  the  geometry.  The  methods  developed  are 
simple  yet  versatile.  They  can  analyze  texture  patterns  for  global  properties,  characterize  linear  feature 
and  characterize  edges.  They  also  provide  an  enlarged  descriptive  vocabulary  for  image  cues. 

Bar  measures  computed  on  the  symmetry  spectrum  give  a  way  to  determine  the 
symmetry/asymmetry  of  a  pattern  about  an  axis  through  the  center  of  the  circular  shaped  region  T  from 
which  they  are  computed.  The  symmetry  bar  measure  is  high  in  a  direction  where  the  pattern  is  sym¬ 
metrical.  The  symmetry  bar  measure  is  low  in  a  direction  where  the  pattern  is  asymmetrical.  If  all  the 
bar  measures  are  large,  the  pattern  is  symmetric  about  the  center  of  the  circle. 

Bar  measures  on  the  contrast  spectrum  reflect  the  Gestalt  principle  of  uniformity  and  proximity. 
This  principle  states  that  close  and  similar  elements  will  be  perceptually  grouped  together.  For  a  given 
d,  the  contrast  measure  indicates  the  similarity  of  gray  levels  d  units  apart.  If  the  patterns  are  similar, 
the  contrast  measure  will  be  large.  A  contrast  bar  measure  is  the  average  contrast  measure  for  a  con¬ 
tiguous  set  of  d’s  in  a  given  direction  while  lie  in  the  circular  spectrum.  For  a  uniform  pattern  the  con¬ 
trast  bar  measure  will  be  large  while  for  a  nonuniform  pattern  the  contrast  bar  measure  will  be  small. 

In  our  calculations  the  spectrums  for  the  contrast  measure  and  the  symmetry  measure  must  be 
computed.  The  coordinates  and  values  of  minima  in  the  spectra  are  determined  and  bar  measures  from 
these  two  spectra  are  computed.  These  entities  can  be  easity  computed  from  the  spectra. 

Let  us  now  consider  how  the  above  principles  can  be  applied  to  edge  detection.  Consider  the 
images  shown  in  Figure  4.  The  direction  of  an  edge  is  traditionally  [Rosenfeld  and  Kak  1982]  defined 
to  be  <( t=arctan(gy/gI).  For  the  vertical  edge  in  Figure  4  the  edge  direction  $  is  0  degrees.  The  contrast 
spectrum  is  shown  in  part  (b)  of  the  figure.  The  darker  the  gray  level  the  lower  the  value  of  the  contrast 
measure.  The  presence  of  an  edge  can  be  determined  by  examining  bar  measures  which  are  perpendic¬ 
ular  to  each  other.  In  this  case  the  vertical  bar  direction  has  the  highest  average  value  of  the  contrast 
measure.  A  horizontal  bar  direction  has  the  lowest  average  value  of  the  contrast  measure.  The  pattern  is 
the  most  uniform  in  the  direction  0=  90  degrees,  Figure  4  (d),  which  corresponds  to  the  maximum  in 
the  contrast  bar  measure.  This  indicates  the  edge  is  in  the  direction  <>=  0  degrees. 

There  is  another  way  to  describe  this  edge.  A  vertical  edge  is  maximally  asymmetric  about  the  y 
axis  and  symmetric  about  the  x  axis.  That  is,  the  pattern  is  symmetric  in  the  8=0  degree  direction.  The 
symmetry  bar  measures  should  provide  another  independent  characterization  of  edges.  Note  that  the 
symmetry  and  contrast  measures  are  independent  in  that  they  measure  different  pattern  characteristics. 
Pan  (b)  of  Figure  4  shows  the  contrast  spectra  and  part  (c)  shows  the  symmetry  spectra.  Part  (d)  shows 
the  contrast  bar  measure  and  part  (e)  shows  the  symmetry  bar  measure.  Note  that  the  vertical  bar  of 
the  symmetry  measure  is  a  maximum  in  the  0=90°  direction.  This  indicates  the  pattern  is  symmetrical 


in  the  direction  0  =  90  degrees.  Note  that  the  minimum  occurs  in  the  0-  0  degrees  which  indicates  the 
pattern  is  asymmetrical  in  this  direction.  The  pattern  is  symmetrical  about  the  x  axis  in  this  case.  The 
edge  direction  $  is  0  degrees  according  to  the  traditional  calculation.  This  is  90  degrees  from  the  max¬ 
imum  in  the  symmetry  bar  measure.  A  calculation  for  a  an  edge  in  any  other  direction  would  proceed 
in  the  same  manner. 

Now  consider  the  comer  region  in  Figure  4.  Note  that  the  symmetry  spectrum  has  a  maximum 
bar  measure  in  the  direction  45  degrees.  The  pattern  is  therefore  symmetrical  in  this  direction.  This 
indicates  the  pattern  is  symmetrical  about  the  1 35  degree  axis  which  characterizes  the  comer. 

Now  let  consider  detecting  the  edge  when  the  difference  between  the  two  regions  is  a  texture 
difference.  Figure  5  shows  an  area  where  an  edge  exists  between  a  forested  area  and  a  grassy  area. 
Note  that  this  difference  is  determined  by  a  symmetry  measure.  The  pattern  is  symmetrical  in  the  0=90 
degree  direction.  This  is  reflected  in  the  bar  measures  extracted  from  the  E2  region.  The  symmetry  bar 
measure  in  the  0=90  degrees  direction  has  the  maximum  value.  This  indicates  the  direction  of  the 
edge  is  <f>=0  degrees.  Note,  that  the  same  methods  work  for  textured  regions  as  previously  we  applied 
to  edges  characterized  by  gray  level  differences. 

Now  consider  the  detection  of  linear  features.  Figure  6.  shows  an  area  with  a  road  structure. 
Observe  that  in  the  region  the  symmetry  bar  measure  is  a  maximum  in  the  0=170  degrees  direction. 
This  indicates  the  pattern  is  symmetrical  in  the  0=170  degree  direction.  This  indicates  the  road  is  in  this 
direction  since  the  road  pattern  is  symmetrical  in  the  direction  of  the  road.  One  might  also  desire  to 
know  the  exact  location  of  the  road.  This  could  proceed  in  a  global  to  local  manner.  In  this  case  one 
knows  the  road  appears  in  the  large  circle.  One  could  proceed  to  examine  smaller  circles  interior  to  the 
large  circle  to  more  precisely  locate  the  road.  Figure  7  shows  three  smaller  regions  extracted  from  the 
same  area.  Part  (a)  shows  the  scene.  Part  (b)  shows  the  contrast  and  symmetry  spectra.  Regions  one 
and  two  are  determined  to  contain  the  road  structure  while  region  three  contains  a  forested  area.  The 
road  direction  is  indicated  by  the  maxima  in  the  bar  measures  shown  in  part  (c).  The  road  is  indicated 
to  be  in  direction  170  degrees  in  R1  and  R2  while  the  R3  bar  measure  indicates  a  road  feature  is  not 
present.  One  could  continue  to  examine  smaller  circles  if  a  more  accurate  location  is  required. 

9.  Developing  a  Hierarchical  Aerial  scene  Analysis  System. 

An  important  problem  related  to  developing  a  vision  system  is  to  have  operators  that  reliably 
return  image  cues  appropriate  for  the  later  phases  of  processing.  It  is  important  to  develop  operators 
whose  measures  have  meaning  to  human  interpreters.  This  implies  that  the  operators  should  measure 
some  understandable  perceptual  property.  Operators  that  measure  perceptual  properties  should  have  a 
certain  robustness.  They  should  preserve  perceptual  similarity.  This  means  that  two  similar  areas  in  the 
scene,  for  example  two  residential  areas,  would  be  close  in  measurement  space.  Thus,  the  vision  system 
should  not  make  severe  errors  such  as  calling  a  residential  area  some  disparate  label  such  as  commer¬ 
cial.  It  is  also  important  to  develop  operators  which  extract  image  cues  at  intermediate  levels  in  the 
scene  hierarchy.  This  will  relieve  the  inference  system  from  undue  dependence  upon  low  level  image 
cues.  An  example  might  be  an  operator  that  examines  an  area  and  determines  if  the  image  patterns  are 
periodic.  Such  a  determination  might  rule  out  commercial  areas  and  indicate  residential  as  a  possible 
interpretation.  The  previous  sections  have  indicated  some  approaches  to  developing  operators  with 
these  properties. 

The  computer  vision  system  we  are  developing  is  a  hierarchical  system  where  each  level  of  the 
hierarchy  corresponds  to  a  particular  level  of  detail  in  the  scene.  At  any  level  the  system  considers  only 
Ki  objects.  The  K0  objects  give  a  gross  classification  of  the  scene  while  classification  at  level  i  ,i>0, 
gives  a  more  detailed  classification  of  the  scene.  The  objects  are  generic  objects  and  attached  to  each 
node  of  the  hierarchy  is  a  frame  or  schema  for  modeling  the  objectfHarlow  et.  al.  86].  A  key  to  the 
development  of  this  system  is  to  have  operators  able  to  characterize  the  object  at  any  level.  Our  desire 
is  to  develop  intermediate  level  operators  for  extracting  image  cues  and  thereby  reduce  the  dependence 
upon  low  level  image  cues.  If  this  can  be  done,  then  every  level  in  the  hierarchy  can  be  treated  simi¬ 
larity.  At  any  given  node  one  can  directly  operate  on  the  scene  data  and  transmit  information  to  nodes 
on  the  level  above  or  below  the  node.  The  frames  at  each  node  model  complex  objects  which  are  in 


turn  composed  of  simplier  objects  forming  a  hierarchy  of  scene  objects.  Nodes  higher  in  the  hierarchy 
obtain  image  information  and  pass  it  down  the  hierarchy.  Because  the  objects  are  generic  objects  they 
can  provide  global  constraints  on  the  scene.  At  each  node  in  the  hierarchy  a  context  is  provided  and 
the  number  of  objects  to  be  located  in  this  context  is  limited,  figure  8.  Also  there  should  be  available  at 
the  node  information  provided  by  the  context  as  to  locations  of  the  subobjects  of  the  object.  This  can  be 
important  in  aerial  scenes  where  the  possible  objects  of  interest  that  can  occur  in  a  scene  may  be  in  the 
hundreds. 

The  system  provides  a  natural  method  to  incorporate  hierarchical  information  into  the  analysis. 
The  hierarchical  information  comes  from  the  interrelationships  given  by  the  nodes  through  the  inheri¬ 
tance  property  of  frames.  The  relational  information  such  as  object- 1  is  left  of  object-2  is  expressed  in 
the  frames.  There  is  an  emphasis  in  this  vision  system  for  gathering  independent  evidence  related  to  the 
presence  of  objects.  This  means  that  one  needs  a  method  for  combining  independent  information 
sources  and  obtaining  a  belief  in  the  presence  of  the  object  based  upon  all  the  available 
information  [Shafer  1976],  A  belief  maintenance  system  is  needed  to  combine  the  information  obtained 
from  the  different  sources.  Several  systems  have  been  proposed  some  of  which  are  heuristic  and  some 
have  a  theoretical  basis.  Fuzzy  logic  and  the  Dempster-Shafer  theory  are  examples  of  systems  with  a 
theoretical  base.  It  is  felt  that  a  reasonable  choice  of  a  belief  maintenance  is  the  Dempster-Shafer  sys¬ 
tem.  We  have  conducted  extensive  investigations  with  this  system  and  with  our  operators  in  the 
analysis  of  aerial  scenes[Harlow  et.  al.  1985,  Conners  et.  al.  1984).  The  results  show  promise  in 
developing  new  insights  into  vision  systems. 
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Figure  1.  A  parallelogram  centered  coordinate  s 
gram. 


The  overage  votue  of  the 
primitive  measure  in  this 


(a)  (b) 


Figure  3.  Part  (a)  A  bar  measure  of  a  primitive  measure  spectrum  is  the  average  value  of  the  prim¬ 
itive  measure  over  a  bar  shaped  region  centered  about  the  origin  of  the  spectrum.  Part  (b) 
Multiple  bar  measures  are  shown. 


vertical  edge 


contrast  bar  measures 
(d) 


symmetry  bar  measures 

(e) 


Edge  detection  using  bar  measures  from  the  contrast  and  symmetry  spectra.  Part  (a) 
shows  a  binary  pattern  with  two  circular  areas  superimposed.  Parts  (b)  and  (c)  show 
images  of  the  contrast(left)  spectrum  and  the  symmetry(right)  spectra  of  the  regions  of 
(a).  Parts  (d)  and  (e)  show  bar  measures  as  a  function  of  orientation.  Part  (d)  shows  the 
contrast  bar  measures  part  (e)  shows  the  symmetry  bar  measures.  The  contrast  and  sym¬ 
metry  bar  measures  have  maxima  aligned  in  the  direction  of  the  vertical  edge.  The  sym¬ 
metry  bar  measure  is  a  maximum  in  the  direction  of  the  symmetry  of  the  comer  region. 
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s  of  Textured  Regions.  Part  (a)  shows  three  areas  E1.E2  and  E3.  E2  is  on  the  boun- 
between  the  forested  regions  and  a  grassy  region.  Part  (b)  shows  the  contrast  and 
ietry  spectrums  for  the  three  regions  starting  with  El  on  the  left.  Pjrt  <c>  shows  the 
leisures  for  the  symmetry  and  contrast  spectrums  for  the  three  regions 
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Figure  6.  Linear  Features.  Part  (a)  shows  a  circular  region  from  which  spectrums  are  computed. 

The  region  contains  a  road.  Part  (b)  shows  Bar  measures  for  the  symmetry  and  contrast 
measures.  The  road  orientation  is  indicated  by  the  maxima  in  the  bar  measures. 


(C) 

Road  Features.  Part  (a)  shows  three  circular  regions.  R1  and  R2  have  a  road  in  the 
region.  Part  (b)  shows  the  contrast  and  symmetry  spectmms.  Part  (c  •  :a .ws  the  bar  meas¬ 
ures  tor  the  contrast  and  symmetry  measures. 
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